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ABSTRACT 


A  conceptual  approach  for  evaluating  and  selecting  among 
alternative  Electronic  Data  Processing  (EDP)  systems  proposed 
to  meet  a  set  of  EDP  user  needs  has  been  developed  by  applying 
cost-effectiveness  methods  and  techniques  to  the  source  selec¬ 
tion  problem.  The  report  provides  a  framework  that  allows  the 
EDP  system  evaluator  to  combine  the  selected  relevant  system 
performance  measures  and  the  related  cost  elements  to  arrive 
at  a  rational  defendable  selection  decision. 
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SECTION  I 


INTRODUCTION 


The  purpose  of  this  technical  report  is  to  document  a  conceptual 
approach  for  evaluating  and  selecting  among  alternative,  proposed 
Electronic  Data  Processing  (EDP)  systems  designed  to  meet  a  set  of  EDP 
user  needs.  The  proposed  approach  was  developed  by  applying  cost- 
effectiveness  methods  and  techniques  to  the  source  selection  problem. 
This  work  was  accomplished  under  Project  8510. 

The  selection  of  EDP  systems  has  presented  a  problem  to  the 
decision-maker  for  many  years  since  competition  became  established  in 
the  production  of  computing  equipment.  The  development  of  computers 
has  been  characterized  by  a  diversity  of  approaches,  designs,  and 
configurations . 

Solutions  to  this  problem  have  been  varied  and  cover  a  wide  spec¬ 
trum  all  the  way  from  essentially  ignoring  the  technical  differences 
to  the  carrying  out  of  detailed  studies  utilizing  sophisticated  tools. 

This  problem  is  not  limited  only  to  the  EDP  user.  The  manufac¬ 
turers  themselves  must  make  design  decisions  involving  a  trade-off 
between  the  cost  of  the  equipment  and  its  performance.  For  a  partic¬ 
ular  user  the  problem  is  basically  simpler  since  he  can  confine  his 
evaluation  to  how  the  EDP  equipment  proposed  by  the  vendor  satisfies 
his  particular  requirements. 

A  significant  amount  of  attention  has  been  devoted  in  the  computer 
literature  to  the  definition  of  measures  of  system  performance  and 
effectiveness.^"^]  These  definitions  have  become  further  complicated 
with  the  availability  of  large-scale  multiple-access  computer  systems. 

In  this  paper,  it  is  sufficient  for  us  to  assume  that  the  user  has 
determined  his  system  requirements  together  with  the  associated  system 
constraints.  It  is  hoped  that  this  paper  will  provide  a  framework  to 
allow  the  EDP  system  evaluator  to  combine  the  selected  relevant  system 
performance  measures  and  the  related  cost  elements  to  arrive  aL  a 
rational  defendable  selection  decision. 

The  approach  adopted  for  selection  itself  must  represent  a  trade¬ 
off  between  effort  (time  and  resources)  applied  and  credibility  achieved 
There  are  some  analysts  who  feel  that  throwing  a  multi-sided  die  may  be 
sufficient  to  select  among  qualified  vendors.  Such  a  process  certainly 
reduces  the  expenditure  of  selection  resources,  but  unfortunately  it 
suffers  in  the  areas  of  repeatability,  defendabil ity ,  credibility,  and 
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acceptance  by  the  competing  vendors  and  the  seleccion  authority  who 
is  responsible  for  the  final  decision, 

A  number  of  EDP  equipment  selection  procedures  have  been  described 
in  the  literature  [17-25]  and  a  far  greater  number  have  undoubtedly  been 
used  but  not  formally  reported.  None  of  the  reported  procedures  have, 
in  the  opinion  of  the  authors  of  this  report,  satisfactorily  handled 
the  problem  of  combining  performance  and  cost.  In  the  last  analysis, 
all  methods  must  make  use  of  an  explicit  determination  of  the  worth* 
to  the  user  of  the  variety  of  features  proposed  by  the  competing  ven- 
dors.  Such  methods  depend  heavily  upon  intuitive  judgment.  It  is  the 
identification  of  rhese  judgment  areas  and  the  degree  to  which  they 
can  be  rendered  explicit  and  defendable  that  contribute  to  the  "success" 
of  a  particular  selection  process. 

In  this  report  we  have  attempted  to  apply  cost-effectiveness 
analysis  techniques  to  the  problem  of  EDP  equipment  selection.  The 
term  cost-effectiveness  has  been  much  maligned  in  the  literature  and 
in  certain  political  circles  in  the  past  few  months.  It  is  not  our 
purpose  to  enter  into  the  socio-political  aspects  of  these  techniques, 
but  rather  to  apply,  to  the  problem  of  EDP  system  selection,  methods 
and  techniques  which  have  proven  superior  in  the  concept  formulation 
and  systems  planning  phase  of  the  systems  acquisition  process. 

jji 

It  should  also  be  pointed  out  that  the  cost-effectiveness  analysis 
techniques  developed  in  this  report  specifically  for  application  to 
EDP  system  selection  are  also  applicable  to  the  source  selection  process 
in  other  system  areas. 

% 

V. 


A  more  detailed  discussion  of  worth  will  be  presented  later  in  the 
report,  in  Section  III. 
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SECTION  II 


OVERVIEW  OF  THE  SELECTION  PROCESS 


In  general,  the  evaluation  and  selection  process  involves  three 
main  components  as  indicated  in  Figure  1 

1.  Statement  of  User  Needs 

2.  Submission  of  Vendor  Proposed  System 

3.  Measurement  or  Comparison  of  the  Proposal 
Against  the  Stated  User  Needs 

Each  of  these  parts  will  be  discussed  in  more  detail  and  various 
terms  will  be  defined  for  later  reference. 


USER  NEEDS 

The  first  task  to  be  performed  is  to  determine  and  make  explicit 
an  approved  set  of  user  needs  which  will  form  the  basis  of  the  Request 
For  Proposal  (RFP)  and  the  evaluation  and  selection  procedure.  For  an 
EDP  system,  the  user  needs  can  be  expressed  mainly  by  the  description 
of  the  future  workload  which  the  user  feels  he  will  have  to  process 
during  the  operational  life  of  the  EDP  system.  The  main  difficulty  in 
expressing  these  needs  is  that  while  the  user  may  be  able  to  express 
accurately  his  current  workload,  he  is  never  really  sure  about  the 
future  workload.  It  is  very  difficult  for  the  user  to  predict  what 
he  may  be  asked  to  process  as  much  as  five  or  more  years  after  the  RIP 
has  been  issued. 

While  user  needs  are  expressed  mainly  in  terms  of  EDP  jobs,  there 
are  other  needs  or  constraints  which  relate  to  the  EDP  system1 s  ability 
to  perform  these  jobs.  One  such  constraint  is  the  maximum  time  allowed 
to  perform  any  one  job  or  a  total  set  of  job^.  For  example,  the.  user 
may  feel  that: 

(a)  he  needs  a  two-second  response  time  for  some  task; 

(b)  some  set  of  jobs  must  be  completed  during  an  eight- 
hour  first  shift  operation; 

(c)  the  monthly  workload  must  be  completed  in  less  than, 
say,  600  hours; 

(d)  the  system  must  be  delivered  within  90  days  after 
contract  award 
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TOTAL  SYSTEM 


Figure  I.  Overview  Of  The  Selection  Process 


4 


It  is  important  to  realize  that  some  of  these  constraints  may  be 
quite  firm  to  the  user;  others  may  be  "open-ended",  i.e.  are  really 
"desires".  Several  factors  complicate  the  problem  of  clearly  stating 
user  needs.  These  factors  are: 

(a)  the  uncertainty  of  the  future  workload, 

(b)  the  lack  of  a  cost  limitation  recognized  by  the 
user  (his  main  pressure  may  be  to  satisfy  the  future 
workload  he  feels  he  will  be  called  upon  to  meet 
independently  of  cost); 

(c)  the  lack  of  cost-sensitivity  information  regarding 
what  it  costs  to  meet  different  combinations  of 
user  needs. 

Because  of  these  difficulties,  some  compromises  must  be  made 
between  the  user  and  the  external  agencies  in  the  procurement  chain, 
such  as  AFADA  and  ESQ,  to  arrive  at  an  approved  set  or  user  needs 
which  will  be  used  as  the  basis  of  the  Source  Selection  Plan. 


PROPOSED  SYSTEM 


Upon  receipt  of  the  RFP  the  vendor  performs  various  cost -performance 
trade-offs  to  configure  an  EDP  system  which  in  the  vendor1 s  opinion  will 
"best"  meet  the  stated  user  needs. 

This  Vendor  System  consists  of: 

(a)  hardware  having  stated  technical  characteristics; 

(b)  software  including  the  various  programs  required 
to  support  operating  systems,  compilers,  etc.;  and 

(c)  vendor  support  including  required  maintenance, 
documentation,  training  of  user  staff,  and  systems 
analysis . 

However,  there  are  other  parts  of  the  Total  System  which  the  user 
must  provide  to  make  the  system  function.  This  User  System  includes 
the  user's  operators,  analysts,  programmers  and  facilities. 


MEASUREMENT 

The  Total  System  as  proposed  by  each  vendor  must  be  compared 
against  the  stated  user  needs,  and  the  winning  vendor  selected  according 
to  specified  criteria.  There  are  two  parts  to  such  an  evaluation: 
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1*  A  system  performance  evaluation  which  determines 
the  effectiveness  of  the  system.  Here  effective¬ 
ness  is  defined*  as  the  degree  to  which  the  system 
will  meet  the  future  workload  and  satisfy  the 
constraints. 

2.  A  cost  evaluation  which  determines  the  total  cost 
for  procurement,  operations  and  maintenance  in 
performing  the  future  workload  over  the  total 
required  operating  life  of  the  system. 

Implicit  in  the  evaluation  is  the  need  to  validate  the  vendor1 s 
proposal.  It  should  be  stated  that  this  requirement  is  common  to 
all  evaluation  procedures  and,  from  a  technical  point  of  view,  may 
represent  the  most  time-consuming  part  of  the  evaluation.  This  area 
will  be  discussed  in  more  detail  below  under  the  subject  of  Vendor 
Uncertainties .  * 


SELECTION  OBJECTIVE 

To  compare  alternative  selection  procedures,  an  explicit  defini 
t ion  of  the  selection  objective  is  required.  Ihe  objective  selected 
is  as  follows: 

MTo  select  a  proposed  EDP  System  which  performs 
a  set  of  future  EDP  jdbs  and  meets  the  job  con¬ 
straints  at  the  Lowest  Total  Cost  to  the  Air  Forcfe, 

*  taking  into  account  Job  Uncertainties  and  Vendor 
Uncertainties 

This  objective  includes  the  following  three  major  concepts: 

1.  All  vendors  must  show  that  their  proposed  systems 
can  perform  the  future  EDP  jobs  and  meet  the  job 
constraints . 

2.  Lowest  total  cost  to  the  Air  Force  should  be  the 
selection  criterion. 

3.  Vendor  and  job  uncertainties  are  the  key  factors 
which  make  the  evaluation  selection  process  a 
difficult  one.  Hence  let  us  now  discuss  the 
various  aspects  of  these  uncertainties. 


*  This  term  will  be  discussed  in  greater  detail  in  Section  III. 
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UNCERTAINTIES 


As  discussed  above,  the  selection  process  is  complicated  by  two 
classes  of  uncertainties  -  vendor  uncertainties  and  job  uncertainties. 
Each  of  these  classes  will  now  be  discussed  in  greater  detail. 

Vendor  Uncertainties 


The  proposal  submitted  by  the  vendor,  in  addition  to  cost  and 
contractual -type  information,  will  include  the  following  technical 
information. 

Technical  Characteristics 


These  are  the  specifications  of  the  components  of  the 
proposed  computer  configuration  together  with  detailed  information 
about  the  performance  of  each  (e.g.  speed,  capacity,  etc.).  Assuming 
that  the  equipment  will  be  delivered  on  time,  one  can  question  whether 
each  component  will  perform  at  the  levels  claimed. 

Software 

In  response  to  the  RFP,  the  vendor  will  describe  those 
software  packages  that  he  will  make  available  with  his  equipment. 
Again,  assuming  that  the  packages  will  be  available  when  needed,  one 
can  question  what  elements  and  functions  are  provided,  how  well  each 
is  implemented,  and  how  well  each  may  be  used. 

System  Performance 

Not  only  must  the  elements  of  hardware  and  software  be 
considered  individually,  but  their  interrelationships  must  also  be 
considered  in  determining  their  effects  on  system  performance.  For 
example,  a  card  reader  or  a  printer  may  not  be  able  to  run  at  rated 
speeds  because  of  other  system  requirements,  or  a  software  package 
may  degrade  system  performance  because  it  produces  inefficient  code, 
because  it  may  constrain  an  operating  program  by  reducing  the  amount 
of  storage  space  available,  or  because  it  is  not  suitably  matched  to 
the  available  hardware. 

Support 

As  mentioned  above,  one  must  always  be  concerned  with 
the  ability  of  the  vendor  to  deliver  his  equipment  and  associated 
software  as  scheduled.  This  is  just  one  example  of  a  number  of 
vendor-dependent  activities  that  can  be  grouped  together  under  the 
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heading  of  vendor  support.  For  example,  the  reliability  of  the 
vendor-supplied  equipment  and  programs  can  very  strongly  affect 
estimates  of  system  timing.  Also,  the  user's  ability  to  operate 
the  vendor's  equipment  will  depend  on  the  documentation  available 
and  on  the  professional  capability  of  the  analysts  and  support 
personnel  provided  by  the  vendor.  Finally,  it  must  be  realized 
that  both  equipment  and  software  must  be  maintained.  The  vendor's 
ability  to  do  this  efficiently  and  systematically  will  also  influence 
the  user's  ability  to  attain  predicted  system  performance. 

Coping  With  Vendor  Uncertainties 

>  } 

A  number  of  techniques  have  been  developed  for  dealing  with 
vendor  uncertainties.  Basically  the  requirement  that  the  vendor 
supply  off-the-shelf  equipment  and  undergo  a  live-test  demo  istration 
removes  a  large  part  of  the  risk  associated  with  making  state-of-the- 
art  systems  operational.  Of  course,  this  procedure  has  a  compensating 
drawback  in  that  it  may  prevent  the  user  from  acquiring  newly  devel¬ 
oped  systems. 

The  available  techniques  may  be  categorized  into  three  major 

areas : 

Professional  Personnel 

The  basic  ingredient  for  any  evaluation  is  the  avail¬ 
ability  of  competent  professional  personnel.  Such  personnel  must  be 
carefully  trained  to  stay  abreast  with  the  state-of-the-art  not  just 
in  the  equipment  alone,  but  also  in  the  way  this  equipment  may  be 
used  such  as  through  time-sharing  or  in  computer  complexes.  The 
ability  of  these  people  to  interpret  and  assess  vendor  claims  will 
be  further  enhanced  through  experience.  In  particular  by  working 
with  vendors,  a  better  understanding  can  be  acquired  of  the  features 
of  the  vendor’s  equipment  and  staff  as  well  as  of  the  marketing 
strategies  employed  by  the  various  vendors.  In  addition,  through 
contact  with  various  Air  Force  user  installations,  a  better  under¬ 
standing  can  be  acquired  of  the  user  needs  and  problems  against  which 
the  vendor  proposals  must  be  assessed. 


Tools 

The  problem  of  validating  vendor  proposals  can  be  greatly 
facilitated  by  having  the  proper  set  of  tools.  Within  the  inventory 
of  applicable  tools,  one  can  identify  the  following  categories: 
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(1)  Simulation  Programs;  ’  J  Basically  it  is  desirable 
to  have  a  program  that  can  take  as  input  a  description  of  the  job  to 
be  performed  together  with  the  specifications  of  the  equipment  proposed. 
The  output  of  the  program  would  be  an  analysis  of  system  performance 
(e.g.  overall  problem  timing,  buffered  times,  component  times,  storage 
requirements,  etc.).  Such  a  program  can  serve  as  a  check  on  the  ven¬ 
dors  logic,  analysis,  and  calculations.  By  incorporating  into  the 
program  an  independently  derived  data  base  for  the  equipment  specifi¬ 
cations,  such  a  program  provides  a  checfc  on  the  vendor* s  data  accuracy. 
Simulation  techniques  are  being  extended  to  evaluate  some  of  the  dynamic 
aspects  of  multiprogramming/multiprocessing  systems. 


(2)  Benchmark  Programs.  The  most  satisfactory  way  to 
validate  a  vendor  proposal  is  to  run  an  actual  live  test.  Because  of 
the  time  and  cost  involved  in  testing  the  whole  job,  one  is  led  to  make 
use  of  a  program  or  set  of  programs  that  are  representative  of  the  job 
to  be  performed  and  constitute  a  predetermined  fraction  of  the  total 
job.  Such  programs  can  be  selected  to  test  the  performance  of  indi¬ 
vidual  computer  components  as  well  as  to  measure,  through  suitable 
extension  factors,  the  overall  system  performance.  Even  though  it  is 
difficult  to  design  such  benchmark  programs  to  test  all  of  the  signif¬ 
icant  aspects  of  the  vendor* s  proposal,  nevertheless  the  use  of  such 
programs  tends  to  restrain  the  vendor  and  to  encourage  his  use  of  more 
defendable  estimates.  In  general,  the  benchmark  programs  are  selected 
as  portions  of  the  actual  expected  workload,  but  programs  already  devel¬ 
oped  for  other  jobs  or  artificially  designed  can  be  used  provided  they 
are  suitably  representative  and  can  be  extrapolated  to  give  the  infor¬ 
mation  desired. 


(3)  Software  Test  Programs.  An  important  class  of  bench¬ 
mark  programs  is  one  especially  designed  for  testing  software.  Here 

it  is  more  efficient  to  design  programs  to  test  specific  elements  or 
combinations  of  elements  of  the  software  package.  However,  a  job- 
oriented  program  may  still  be  useful  to  test  such  features  as  compiler 
efficiency.  There  is  a  modest  amount  of  cooperative  effort  being 
expended  under  the  direction  of  USASI  (U.S.A.  Standards  Institute)  to 
develop  such  compliance  test  programs  for  COBOL. 

(4)  EDP  Data  Base.  Information  concerning  the  avail¬ 
ability  and  performance  of  EDP  equipment  can  be  organized  into  a  data 
base  to  facilitate  the  validation  of  vendor  proposals.  Not  only  does 
this  provide  a  reservoir  of  information  to  check  figures  against,  but 
it  also  serves  as  a  repository  for  cataloguing  acquired  experience 
with  vendor  equipment  and  claims.  As  time  allows,  one  can  envisage 

a  sort  of  "Good  Housekeeping**  approach  to  test  the  performance  of 


9 


computer  components  and  software  packages  wLth  the  test  results  being 
incorporated  into  the  data  base, 

(5)  Analysis/Synthesis .  In  support  of  any  validation 
procedure,  there  must  exist  a  basic  understanding  of  the  performance 
of  the  individual  computer  components  and  the  interrelations  that 
govern  how  these  components  work  together  as  part  of  the  overall 
computer  system.  For  example,  the  performance  of  the  CPU,  storage 
devices,  I/O  devices,  data  channels,  file  structures,  scheduling 
disciplines,  etc.,  must  be  analyzed  in  order  to  predict  the  overall 
system  performance  or  to  determine  those  elements  that  may  be  criti¬ 
cal  to  that  performance.  Such  prediction  and  estimation  must  take 
into  consideration  dynamic  conditions  that  characterize  the  on-line 
use  of  computers  today. 

Systematic  Procedures 

Because  of  the  large  number  of  parameters  that  contribute 
to  the  overall  complexity  of  validating  vendor  proposals,  systematic 
procedures  must  be  established  to  provide  an  orderly  context  for 
assessing  the  vendor3 s  proposal.  Given  a  competent,  professional 
staff  with  an  appropriate  set  of  tools,  it  is  still  necessary  to 
establish  an  unambiguous  set  of  procedures  to  assure  that  the  vendor 
understands  the  user's  requirements,  and  that  the  evaluators  under¬ 
stand  each  vendor's  proposal.  The  user's  requirements  can  be  formal¬ 
ized  into  a  set  of  system  specifications  which  can  be  translated  with 
the  cooperation  of  the  evaluation  team  into  the  Request  For  Proposal 
(RFP)  that  is  transmitted  to  the  vendor.  By  carefully  establishing 
the  format  and  contents  of  the  RFP,  the  vendor  will  know  what  to  expect 
and  what  to  look  for  in  the  RFP.  By  establishing  lines  of  communica¬ 
tion  between  the  vendor  and  the  user/evaluation  team,  the  vendor  can 
Inform  the  team  of  critical  areas  in  his  proposal  and  can  receive  clari¬ 
fication  of  any  questions  on  the  RFP  that  may  arise.  By  following  sys¬ 
tematic  procedures,  one  can  assure  that  relevant  information  is  equitably 
disseminated  to  all  competing  vendors.  Records  can  be  maintained  to 
determine  what  information  was  exchanged  in  case  of  misunderstandings 
that  may  later  arise.  By  applying  established  validation  procedures 
and  evaluation  techniques,  one  can  increase  the  probability  that  the 
vendors  will  accept  the  results  of  the  validation  and  evaluation  exer¬ 
cises.  In  addition,  as  new  techniques  or  improvements  are  developed, 
they  can  be  more  readily  incorporated  into  the  established  procedures. 
Finally,  by  having  an  established  chain  of  approvals  for  the  selection 
plan  and  decisions,  one  has  available  a  set  of  checks  and  balances  that 
will  assure  the  vendor  of  equitable  treatment  and  avoid  the  aura  of 
mistrust  which  might  otherwise  becloud  the  vendor/evaluator  relation¬ 
ship. 
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Job  Uncertainties 


As  was  discussed  above  in  Section  II,  a  number  of  factors 
contribute  to  the  difficulty  of  explicitly  stating  the  user's  needs. 
For  example,  given  that  the  user  will  be  asked  to  perform  a  certain 
job  in  the  future,  a  number  of  aspects  of  that  job  may  change  in  the 
future  and  be  difficult  to  predict  at  the  present  time.  The  size  of 
the  job  may  vary  due  to  changes  in  the  lengths  of  the  files  to  be 
processed  (e.g.  the  number  of  fields  in  a  record  or  the  number  of 
records  might  change).  The  frequency  of  running  certain  jobs  may  be 
difficult  to  predict  and  consequently  the  total  time  demanded  for 
that  job  becomes  uncertain.  Complexity  of  jobs  may  increase  through 
the  incorporation  of  additional  processing  steps  into  the  job  as 
experience  and  requirements  evolve,  or  through  the  introduction  of 
more  refined  or  sophisticated  methodology.  Finally,  the  set  of  jobs 
to  be  performed  may  change  by  the  addition  or  substitution  of  new 
jobs  that  were  not  anticipated  when  the  user  originally  specified  his 
needs . 


The  level  of  credibility  of  the  user's  predictions  of  future 
workload  can  be  raised  by  applying  more  time  and  resources  to  the 
analysis  of  the  user's  requirements.  For  example,  if  a  particular 
selection  involved  a  specified,  approved  workload  to  the  exclusion  of 
unanticipated  future  jobs,  then  through  the  use  of  detailed  systems 
analysis  and  extensive  system  simulation,  one  could  very  accurately 
establish  the  characteristics  of  even  a  complex  workload.  However, 
in  most  cases  there  is  insufficient  time  or  manpower  or  budget  to 
permit  the  extensive  analysis  required  for^accurate  workload  predic¬ 
tion.  In  addition,  user  jobs  do  change  over  time  to  a  degree  which 
may  be  difficult  to  predict. 

Coping  With  Job  Uncertainties 

Recognizing  that  the  future  workload  cannot  be  considered 
fixed  and  completely  specified,  evaluators  have  devised  a  number  of 
techniques  to  cope  with  these  uncertainties.  The  most  commonly 
adopted  method  makes  use  of  a  "point -scoring"  procedure  that  estab¬ 
lishes  a  hierarchy  of  factors  or  criteria  together  with  appropriate 
formulas  and  weights .  1^3 , 28]  Points  are  then  allocated  in  accordance 
with  how  well  each  vendor  has  scored  on  the  various  factors  and  upon 
the  relative  weights  allocated  by  the  evaluation  team  to  these  factors. 
The  vendor  with  the  largest  total  score  is  then  adjudged  to  be  the 
winner.  A  detailed  presentation  of  such  a  procedure  for  selecting 
among  alternatives  has  been  carried  out  by  Dr.  James  R.  Miller. l^J 
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The  remainder  of  this  report  will  be  concerned  with  the 
application  of  cost-effectiveness  analysis  techniques  to  the  problem 
of  EDP  equipment  selection.  It  is  felt  that  the  adoption  of  a  cost- 
effectiveness  approach  provides  a  more  rational  and  equitable  basis 
for  comparing  competing  vendor  proposals  and  for  coping  with  the 
uncertain  workload.  Further  remarks  on  the  advantages  of  the  cost- 
effectiveness  approach  will  be  made  in  Section  IV. 


t 
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SECTION  III 


COST-EFFECTIVENESS  ANALYSIS  APPROACH 


With  the  previous  statement  of  the  problem  and  the  various 
evaluation  difficulties  as  background,  we  shall  now  discuss  the 
approach  taken  in  applying  conventional  cost-effectiveness  analysis 
to  this  problem  of  computer  evaluation  and  selection 


DEFINITIONS 

Let  us  start  the  discussion  by  defining  various  terms  which 
will  be  applied  in  the  analysis  * 

1.  Effectiveness :  The  degree  to  which  a  system  will  perform 
the  future  jobs  and  satisfy  the  constraints.  Effectiveness  is 
generally  considered  to  consist  of  the  following  three  main  compo¬ 
nents  as  illustrated  in  Figure  2: 

a.  Capability :  The  degree  to  which  a  system  will  perform 
the  future  jobs  and  satisfy  the  constraints,  assuming  that  the  system 
is  always  available  for  operation  and  will  never  malfunction.  Capa¬ 
bility  can  be  measured  in  various  ways,  but  the  two  key  measures  of 
capability  are. 

(1)  Quality  of  the  work  output.  This  measure  is  in 
general  multi -dimensional  since  it  encompasses 
the  many  sub-measures  used  to  measure  the  work 
output.  For  example,  it  might  include  the 
straightness  of  a  line  of  print  or  the  maximum 
number  of  copies  of  printout. 

(2)  Time .  Given  that  the  quality  of  the  work  output 
can  be  measured,  a  second  key  measure  of  the  EDP 
system  capability  is  the  time  taken  to  p  erfcrm 
the  future  workload,  again  assuming  the  system 

is  available  for  operation  and  never  malfunctions. 
For  example,  the  measure  could  be  the  expected 
time  to  perform  a  given  monthly  workload. 


These  concepts  are  an  application  of  those  used  in  the  evaluation 
of  weapon  systems.  Reference  is  made  to  such  reports  as  "Weapon 
System  Effectiveness  Industry  Advisory  Committee  (WSEIAC) ,  Final 
Summary  Report,  AFSC-TR-6 5-6 ,  January  1965. 
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EFFECTIVENESS 


1. 


CAPABILITY  TO  PERFORM 
JOBS  &  SATISFY  CONSTRAINTS 


TIME 

QUALITY 


2.  AVAILABILITY 


3.  DEPENDABILITY 


NON-PRODUCTIVE  TIME 

DOWN  TIME  (SCHEDULED/UNSCHEDULED) 
LOST  TIME  (DUE  TO  ERRORS) 


COST 

TOTAL  DOLLARS  REQUIRED  TO  PERFORM  THE  FUTURE  SET  OF  EDP  JOBS 


Figure  2.  Cost-Effectiveness  Definitions 


b.  Availability  may  be  defined  as  the  probability  that  the 
system  will  be  ready  for  operation  when  called  upon. 

c.  Dependability  may  be  defined  as  the  probability  of  the 
system  completing  the  job  satisfactorily,  given  that  it  was  avail¬ 
able. 


However,  when  evaluating  EDP  systems  in  which  the  primary 
measure  of  capability  is  the  expected  time  to  perform  a  given  work¬ 
load  (given  that  the  quality  of  the  system  meets  a  certain  level  of 
acceptability),  both  availability  and  dependability  can  correspon¬ 
dingly  be  measured  in  units  of  non-productive  time  expected  during 
the  performance  of  a  given  (monthly)  workload.  This  non-productive 
time  consists  of  two  sources: 

(1)  Down  time:  both  scheduled  maintenance  to  prevent 
malfunctions  and  unscheduled  maintenance  to  detect 
and  correct  malfunction. 

(2)  Lost  time:  non-productive  run  time  requiring  rerun 
because  of  suspected  or  detected  errors. 

Availability  also  includes  how  well  the  vendors  can  meet  the 
desired  delivery  date,  since  this  has  an  impact  on  the  non-productive 
time  of  the  system, 

2.  Cost  The  total  dollars  required  to  procure,  operate,  and 
maintain  the  system  to  perform  the  future  set  of  EDP  jobs.  As  indi¬ 
cated  previously  in  Figure  1,  all  costs  are  to  be  included  in  making 
this  computation  for  both  the  vendor  and  the  user.x 


THE  SELECTION  PROBLEM 

Given  that  the  effectiveness  and  cost  of  a  particular  system 
can  each  be  measured  separately,  the  evaluation  team  will  still  be 
faced  with  the  problem  of  how  to  combine  these  two  factors  to  reach 
a  final  selection.  For  example,  as  illustrated  in  Figure  3,  we  might 
have  a  situation  where  System  B  provides  a  higher  level  of  effective¬ 
ness  than  System  A  but  costs  more.***  The  source  selection  problem 
is,  "Which  is  the  better  system  to  buy?"  This  could  be  restated  as, 


In  some  cases  certain  non-differentiating  costs  may  be  omitted. 

Note  that  the  decision  is  straightforward  if  we  have  a  "dominant" 
case  where  one  system  provides  more  effectiveness  at  a  lower  cost. 
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Figure  3.  Cost-Effectiveness  Analysis 
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MIs  the  additional  amount  of  effectiveness  worth  the  added  amount  of 
cost?"  It  is  impossible  to  answer  this  question,  except  on  a  purely 
intuitive  basis  (which  may  be  wrong  or  difficult  to  defend),  without 
resorting  to  either  of  the  two  source  selection  criteria  used  in  a 
cost-effectiveness  analysis: [31] 

1.  Specify  a  level  of  effectiveness  which  all  systems  must 
meet,  and  select  that  system  which  meets  this  level  at 
lowest  total  cost.  This  criterion  is  called  "Pivoting 
on  Constant  Effectiveness" .  Thus,  if  E2  is  chosen  as 
the  comparison  level  of  effectiveness  and  the  effective¬ 
ness  of  System  A  is  increased  accordingly,  its  new  "Opera¬ 
ting  Point"  on  Figure  3  might  be  either  at  (lower 

cost  than  B  and  hence  selected),  or  A^  (higher  cost 
than  B  and  hence  rejected). 

2.  Specify  a  level  of  cost  which  all  systems  must  not  ex¬ 
ceed,  and  select  that  system  which  provides  the  highest 
level  of  effectiveness.  This  is  called  "Pivoting  on 
Constant  Cost".  Thus,  if  C2  is  chosen  as  the  comparison 
level  of  cost,  and  the  cost  of  System  A  is  increased 
accordingly,  its  new  "operating  point"  on  Figure  3  might 
be  either  at  A2  (higher  effectiveness  than  B  and  hence 
selected)  or  A^  (lower  effectiveness  than  B  and  hence 
rejected) . 

The  first  selection  criterion  will  be  used  for  illustration  for 
the  remainder  of  this  report,  since,  in  general,  government  procure¬ 
ment  uses  this  criterion. 


PROPOSED  SELECTION  PROCESS 

We  shall  now  discuss  in  greater  detail  the  three  categories 
which  we  have  used  to  describe  user  needs,  a  method  for  quantita¬ 
tively  communicating  these  needs  to  the  vendors,  and  a  procedure 
for  evaluating  how  well  each  vendor1 s  proposed  system  meets  each 
of  these  three  categories  of  needs. 

Classification  of  User  Needs 


Taking  into  account  the  many  jobs  which  could  make  up  a 
future  workload  and  the  various  uncertainties  associated  with  each 
job  and  with  the  vendors*  proposals,  efforts  were  made  to  apply 
cost-effectiveness  analysis  methods  and  techniques  to  the  source 
selection  process  of  Figure  1.  The  analytical  approach  used  was 
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concentrated  on  making  explicit  all  of  the  characteristics  of  the 
possible  jobs  and  on  quantifying  the  uncertainties  associated  with 
each  job. 


It  Loon  became  apparent  that  this  might  not  be  practical  to 
do  on  every  source  selection  since  some  might  require  an  excessive 
expenditure  of  time  and  user/analyst  manpower.  Hence  it  was  decided 
to  restructure  the  user  needs  portion  of  the  selection  process  as 
shown  in  Figure  4.  User  needs  can  be  considered  to  be  made  up  of 
three  primary  parts.  The  first  part  describes  the  representative 
workload.  The  second  and  third  parts  which  were  formerly  encom¬ 
passed  by  the  term  "Constraints”  will  be  more  explicitly  defined  as 
mandatory  requirements  and  desirable  features. 

a .  Representative  Workload 

Out  of  the  many  jobs  which  the  user  predicts  may  make  up 
the  future  workload,  only  the  most  important  or  a  selected  subset 
would  be  used  to  represent  this  future  workload  by  adjusting  job  types 
and  their  frequencies  of  operation.  Jobs  again  consist  of  "Known  Jobs", 
for  which  the  user  has  a  high  degree  of  confidence  in  their  occurring 
and  in  their  characteristics,  such  as  size  and  frequency  of  operation, 
and  "Likely  Jobs",  for  which  the  degree  of  confidence  is  lower.  An 
analytical  technique  for  expressing  this  uncertainty  by  making  use  of 
quantitative  probabilistic  estimates  will  be  described  in  Section 
III. 


b.  Mandatory  Requirements 

These  are  absolute  requirements  which  must  be  met  if  a 
system  is  to  be  considered  for  further  evaluation.  A  system  that  does 
not  satisfy  one  or  more  of  these  requirements  is  considered  non-responsive 
and  is  essentially  disqualified.  Since  this  is  such  a  strong  constraint, 
every  effort  will  be  made  to  keep  these  requirements  to  a  minimum. 

c.  Desirable  Features 


This  third  category  is  required  to  express  user  needs 
because  of  the  following  reasons: 

(1)  As  indicated  previously,  the  representative  workload 
only  approximates  the  actual  expected  workload.  Since  there  may  be 
other  requirements  of  the  workload  that  will  not  be  measured  in  the 
system  timing  determination,  inclusion  of  selected  desirable  features 
allows  the  user  to  account  for  the  additional  capability  provided  by 
these  features  to  satisfy  the  above  additional  requirements.  In  a  sense, 
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TOXAI.  SYSTEM 


Figure  4.  Proposed  Selection  Process 


19 


these  features  offer  the  user  a  "hedge11  against  uncertainty  in  his 
statement  of  the  expected  workload. 

(2)  Since  all  measurement  techniques  have  some  uncer¬ 
tainty  connected  with  them,  the  inclusion  of  a  list  of  desirable 
features  also  serves  as  a  hedge  against  this  uncertainty  in  measuring 
a  vendor :s  ability  to  meet  the  future  workload. 

(3)  Many  times  a  user  need  which  is  called  "mandatory" 

Ld  ♦'eally  only  a  "desire".  For  example,  is  a  two-second  response 
timt  for  a  task  absolutely  mandatory,  or  would  2,1  seconds  be  per¬ 
missible  for  a  system  which  is  20%  less  expensive.'  it  is  better  to 
list  non  discriminatory  "design  goals"  to  the  vendor  in  this  category 
rather  than  in  mandatory  requirements,  since  elimination  of  a  vendor 
foe  slight  non-responsiveness  (at  large  decreases  in  cost)  may  not 
really  be  defendable. 

(4)  As  long  as  the  production  of  computing  equipment 
continues  to  be  competitive  with  the  introduction  by  different  vendors 
of  improved  design  features,  the  user  is  forced  to  consider  the  benefits 
to  be  obtained  from  these  features.  If  the  availability  of  these  fea¬ 
tures  wlII  be  of  benefit  to  the  user  and  if  those  benefits  are  not  ade¬ 
quately  incorporated  in  the  system  timing  determination,  some  means 
should  be  provided  in  the  selection  process  to  account  for  them. 

The  Representative  Workload 

a .  Descr  ipt  ion 

We  shall  now  discuss  a  method  for  explicitly  describing 
the  representative  workload.  Ftiis  description  will  be  used  by  the 
vendors  in  preparing  their  system  proposals  and  will  be  used  as  a 
basis  for  evaluating  the  performance  of  each  vendor. 

In  explicitly  describing  the  representative  workload, 
the  analyst  must  deal  with  the  uncertainties  that  may  exist  for  each 
poll.*  of  time  in  the  future.  This  can  be  treated  as  two  basic  prob¬ 
lems  first,  there  is  the  estimation  of  the  user’s  workload  for  a 
particular  future,  time  including  a  quantification  of  the  uncertain¬ 
ties  in  this  workload  expressed  in  probabilistic  form;  secondly, 
there  is  che  esnmation  of  how  the  workload  may  change  with  time  in 
the  future  which  may  be  based  on  extrapolations  of  current  and  past 
workload  data. 

(1)  Probabilistic  Workload  Description.  To  express  this 
uncertainty  analytically,  the  user  is  asked  to  define  for  a  given  point 
m  t Lme  a  reference  workload  and  to  provide  a  quantitative  estimate  of 
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the  probability  P  that  the  actual  workload  occurring  at  this  time  may 
exceed  the  specified  reference.  This  can  be  expressed  by  selecting 
certain  multiples  of  the  reference  workload  and  having  the  user  specify 
the  probability  that  the  actual  workload  at  the  selected  time  will  be 
equal  to  or  exceed  each  multiple  of  the  reference  workload.  For  example, 
in  Figure  5  the  user  has  stated  that  for  a  particular  time  there  is  a 
probability  equal  to  1.0  that  the  future  workload  will  exceed  the 
reference  workload,  and  a  probability  P2  equal  to  0.80  that  it  will 
exceed  1,1  times  the  reference  workload,  etc.  Several  comments  can  be 
made  about  such  an  estimate: 

(a)  These  are  the  user’s  subjective  estimates  and  will  have  to 
be  justified  to  higher  level  authorities  such  as  AFADA  and 
Hq.  USAF. 

(b)  While  theoretically  there  may  be  some  small  probability  that 
the  actual  workload  will  exceed  the  upper  bound  shown,  e.g. 

2.0  times  the  reference  workload,  the  probability  can  be  taken 
as  zero  if  it  can  be  mutually  agreed  that  this  will  be  taken  as 
the  practical  upper  design  limit  for  the  EDP  system. 

(c)  While  we  have  described  a  situation  where  a  workload  may  vary 
in  size,  it  may  also  vary  in  complexity  or  in  any  fashion 
which  the  user  chooses  to  make  explicit  and  include  in  his 
estimate.  The  probabilistic  description  presented  above  can 
bo  used  for  each  element  of  such  a  workload  specification. 

The  workload  discussed  below  will  consist,  where  appropriate, 
of  combinations  of  such  elements. 

(j)  The  units  for  expressing  workload  will  depend  upon  the 

particular  situation  in  question.  In  general,  some  common 
measure  such  as  equivalent  machine  time  will  be  used. 

(e)  Since  the  user  has  provided  the  analyst  in  the  example  illus¬ 
trated  in  Figure  5  with  four  estimates  of  workload  levels,  the 
entire  workload  range  may  be  divided  into  four  segments  of  in¬ 
terest  (viz,  S^,  S2,  S«5  S^)  as  shown  in  the  figure.  To  obtain 
estimates  for  intermediate  workload  levels,  the  original  four 
estimates  have  been  connected  by  straight  lines.  If  greater 
estimation  accuracy  is  desired,  additional  estimates  would 
have  to  be  provided  by  the  user. 
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Figure  5.  Probabilistic  Workload  Description  For  A  Given  Time 


(2)  Workload  Growth  With  Time.  As  indicated  previously, 
workloads  change  and  generally  grow  with  time.  Hence  a  probabilistic 
estimate  of  the  representative  workload  is  needed  for  various  periods 
of  time.  The  summation  of  this  data  may  be  structured  as  a  function 
of  operational  year  as  shown  in  Figure  6a.  In  the  example  shown,  a 
total  operational  life  of  five  years  and  a  linear  growth  of  workload 
with  time  are  assumed.  Obviously,  this  linear  workload  over  time  is 
only  an  approximation  to  the  real  situation  expected  which  may  vary 
irregularly  due  to  seasonal  or  irregular  demands.  However,  even  this 
approximation  to  the  actual  demand  function  can  serve  as  a  design 
guide  and  evaluation  measure. 

In  Figure  6a,  each  workload  line  has  been  assigned 
a  probability  level  corresponding  to  the  levels  selected  in  Figure  5. 
For  example,  the  lowest  line  in  Figure  6a  represents  a  workload  as  a 
function  of  time  which  the  user  has  indicated  has  a  100%  probability 
of  being  experienced  or  exceeded.  In  other  words,  the  user  has  speci¬ 
fied  that  he  is  completely  certain  that  his  workload  will  be  at  least 
as  great  as  the  amount  shown  by  this  lowest  line. 

The  lowest  line  in  Figure  6a  has  also  been  labeled 
as  the  "Reference  Workload".  The  specification  by  the  user  of  his 
reference  workload  is  independent  of  the  probability  level  he  attaches 
to  that  level.  In  other  words,  the  user  will  first  specify  his  ref¬ 
erence  workload  by  whatever  predictive  tools  he  has  available.  Then, 
through  an  independent  operation,  he  will  assign  a  probabilistic  level 
to  that  workload. 


An  even  coarser  approximation  to  the  actual  workload 
is  shown  in  probabilistic  form  in  Figure  6b.  Here  the  average  work¬ 
load  for  the  year  is  used  for  each  entire  year,  so  that  the  workload 
increases  in  discrete  amounts.  As  will  be  seen  later,  such  an  approxi¬ 
mation  simplifies  the  calculations. 

b .  Calculation  of  Total  Expected  System  Cost 

By  constructing  an  explicit  demand  function.  i,e.  the 
probabilistic  workload,  the  user  has  stated  the  range  of  possible 
workloads  for  which  he  is  concerned.  Every  vendor  must  show,  by 
various  means  open  to  him,  that  his  EDP  system  can  meet  any  workload 
up  to  the  indicated  maximum  which  the  user  may  require.  These  means 
may  include  equipment  expansion  or  replacement  at  a  later  time.  It 
may  also  include  the  use  of  service  bureau  leasing  or  satellite  opera¬ 
tion  if  this  is  acceptable  to  the  user. 
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Figure  6b.  Probabilistic  Workload  Description  Averaged  Per  Year 


24 


Since  the  vendor  will  be  provided  with  the  user's  esti¬ 
mates  of  the  predicted  workload,  he  may  now  perform  various  cost- 
performance  trade-off  analyses  resulting  in  a  proposal  of  an  initial 
system  installation  together  with  system  growth  when  and  if  the  actual 
workload  reaches  certain  levels.  The  vendor  costs  for  the  proposed 
initial  system  and  its  growth  will  also  be  provided  in  the  proposal. 

Based  on  this  information,  the  expected  total  cost  of 
meeting  the  probabilistic  workload  can  be  calculated.  The  example 
shown  in  Figure  7  illustrates  the  hypothetical  response  of  Vendor  A 
to  the  workload  described  in  Figures  5  and  6.  This  vendor  has  pro¬ 
posed  to  install  initially  a  system  which  can  perform  the. workload 
for  the  first  two  years  of  operation  within  a  stated  mandatory  require¬ 
ment  of  less  than  600  hours  (allowing  the  remaining  time  for  scheduled 
and  unscheduled  maintenance  as  well  as  any  necessary  reruns  of  errors). 
However,  during  subsequent  years,  the  probable  increase  in  workload  may 
exceed  the  600  hours  available  on  system  A^.  In  fact,  based  on  the 
stated  workload,  there  is  a  100%  chance  that  this  will  occur  in  year  5, 
if  it  did  not  occur  sooner.  To  cope  with  this  increase,  the  vendor 
has  proposed  that  his  initial  system  A-^  be  altered  through  addition  or 
replacement  to  a  new  system,  called  system  A2,  which  can  perform  the 
increased  workload  within  the  600  hour  limitation  and  which  will  be 
available  when  the  user  so  directs.  The  validated  timings  for  each 
system  to  perform  the  different  workload  levels  are  shown  in  Figure  7. 
The  vendor  also  provides  cost  information  indicating  his  proposed  costs 
for  all  elements  of  each  system,  i.e.  A^  and  A2,  as  a  function  of  system 
running  time.  Such  cost  information  would  include  shift  costs  if  rele¬ 
vant  as  well  as  lease  vs.  buy  information. 

To  these  costs  which  the  vendor  would  provide,  the  cost 
analyst  would  add  the  costs  which  the  user  would  incur  in  operating 
the  system  over  the  total  operational  life.  Based  on  this  total  cost 
data,  the  total  expected  cost  C  for  operating  each  vendor's  proposed 
system  can  be  calculated  for  each  year  from  the  formula 

C  =  +  P2C2  +  ••••  +  Pncn  (1) 

where  p^  is  the  probability  that  the  actual  workload  will  be  con¬ 
tained  in  the  segment  and  incur  a  total  cost  ,  and  n  is 

the  total  number  of  segments  used  to  represent  the  workload  range  for 
that  year.  These  segments  can  be  determined  by  the  analyst  based  on 
the  user's  description  of  his  workload. 

In  the  example  we  have  selected,  the  probability  that  the 
actual  workload  will  fall  within  any  one  segment  is  found  simply 
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Figure  7.  Vendor  Response  To  Perform  Representative  Workload 
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[32] 

by  taking  the  difference  between  the  two  cumulative1  J  probabilities 
that  bound  that  segment.  For  example,  the  probability  that  the  work¬ 
load  will  fall  within  the  segment  S3  is  the  difference  between  the 
probability  that  the  workload  will  equal  or  exceed  the  reference  work¬ 
load  (1.0)  and  the  probability  that  the  workload  will  equal  or  exceed 
1.1  times  the  reference  workload.  Referring  to  Figure  5,  the  prob¬ 
ability  that  the  workload  will  fall  within  segment  S3  is  p^  =  0.20 
(viz.  1.00  -  0.80).  Similarly,  we  could  determine  the  probability  p^ 
that  the  workload  will  fall  within  each  of  the  other  segments  S3  . 

These  probabilities  pj-  (i  =  1,  ...,  4  in  this  example) 
can  then  be  used  in  equation  (1)  to  determine  the  total  expected  cost 
for  that  operational  year.  The  determination  of  the  cost  C3  to  be 
applied  to  each  segment  will  depend  upon  the  amount  of  information 
available  to  the  analyst  and  the  accuracy  that  the  analyst  requires 
in  his  calculation.  For  example,  referring  to  Figure  7  for  vendor 
system  there  is  a  probability  of  0.20  that  the  actual  workload 
will  fall  within  segment  S3  ;  i.e.  between  260  hours  and  320  hours. 

If  we  can  assume  that  the  probability  is  distributed  uniformly  within 
this  range  of  workload  and  that  the  cost  is  proportional  to  the  work¬ 
load,  then  we  can  determine  C3  as  the  cost  for  system  A3  to  perform 
an-  average  workload  of  290  hours. 

Similarly,  it  can  be  seen  from  Figure  7  that  there  is 
a  probability  P2  =  0.30  that  the  actual  workload  will  fall  within 
segment  S2  ,  between  320  hours  and  370  hours,  for  vendor  system  A3. 
Again,  assuming  it  is  permissible  to  use  the  average  workload,  cost  C2 
will  be  determined  for  vendor  system  A3  to  perform  the  average  work¬ 
load  of  345  hours.  In  a  similar  fashion,  probabilities  P3  and  p^ 
together  with  costs  C3  and  C4  can  be  determined.  Equation  (1) 
could  now  be  used  to  determine  the  expected  cost  of  operation  for  the 
first  year,  viz.  P3C3  +  P2^2  +  P3C3  +  P4C4  -  same  procedure  could 

then  be  applied  to  each  operational  year  to  determine  the  expected  cost 
for  that  year. 

Several  comments  regarding  this  method  should  be  made: 

(1)  System  discontinuities  may  occur  inside  a  segment. 

For  example,  the  vendor  may  indicate  a  shift  from  system  A3  to  A2 
in  S3  3  ,  the  third  segment  of  the  third  operational  year.  Hence  to 
calculate  properly  the  expected  cost  of  the  third  year*s  operation, 
a  separate  calculation  for  each  of  the  two  sub-segments  must  be  made. 

In  the  above  example,  the  probability  that  the  workload  will  lie 
within  each  sub-segment  would  be  determined  from  a  further  analysis 
of  the  cumulative  distribution  function  of  Figure  5  reproduced  again 
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in  Figure  8.  If  an  assumption  Is  made  that  the.  cumulative  distribu¬ 
tion  function  of  Figure  8  consists  of  straight  line  segments  which 
connect  the  estimates  provided  by  the  user,  then  linear  interpolation 
mav  be  employed.  For  example,  the  probability  that  the  actual  work¬ 
load  is  between  1,0  and  1,05  times  the  representative  workload  i3 
0,10.  A  similar  breakpoint  or  sub-segment  would  be  used  to  represent 
other  discontinuities  that  may  occur  in  system  costs  such  as  might 
accompany  shift  changes.  Again  linear  interpolation  could  be  used  to 
determine  the  probability  associated  with  each  of  the  resultant  sub- 
segments  , 


(2>  Non-productive  time,  previously  discussed  under 
system  availability  and  dependability,  can  be.  handled  in  several 
ways.  The  method  previously  described  assumed  that  tne  reliability 
of  all  vendor  systems  cannot  be  differentiated  from  one  another, 
since  they  will  all  meet  certain  minimum  standards.  Hence  an  arbi¬ 
trary  maximum  productive  time,  e. g.  600  hours,  may  be  chosen  for  all 
vender-,  and  the  vendor  system  expansions  could  all  be  based  on  this 
upper  1 Lmit .  On  the  other  hand,  the  evaluators  could  permit  each 
vendor  to  calculate  his  maximum  productive  time,  and  use  this  figure 
for  expansion  design  purposes.  Obviously,  this  method  will  require 
validation  of  vendor  claims,  but  this  could  be  done  by  using  vendor  - 
claimed  reliability  as  part  of  the  contract,  if  the  vendor  would 
agree  to  do  so.  The  burden  of  proof  for  such  validation  is  still 
on  the  vendor. 

(3)  Note  that  total  system  time  was  used  to  describe 

each  workload  for  which  the  corresponding  co3t  element  was 

determined.  In  an  actual  case,  the  time  corresponding  to  the  uti¬ 
lization  of  each  equipment  component  in  the  proposed  configuration 
would  be  determined  depending  upon  the  vendor* s  cost  elements. 

Again,  depending  upon  the  information  available  and  the  accuracy 
desired,  the  analyst  could  introduce  simplifying  assumptions  to 
keep  the  calculations  tractable  and  commensurate  with  the  e.valua- 
tion  model  selected. 

(4)  After  the  expected  cost,  for  performing  eacn  year:s 
work.  >ad  is  calculated,  we  must  still  combine  this  riCost  Stream* 
over  time  into  one  total  expected  cost.  This  can  be  done  using  the 
standard  approach  of  obtaining  the  total  present  worth  by  reflecting 
each  of  these  costs  back  to  time  zero  using  an  appropriate  interest 
rate.  In  the  same  fashion,  the  lease  vs.  buy  calculation  may  be 
performed  to  determine  which  present  worth  is  the  lower  cost. 
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igure  8.  Probabilistic  Workload  Showing  Linear  Interpolation 


Benefits 


c  . 


Let  us  now  indicate  some  of  the  benefits  of  the  pro¬ 
posed  method  of  specifying  the  workload  in  probabilistic  form  as 
contrasted  with  a  deterministic  method  of  specifying  workload.  The 
deterministic  procedure  requires  that  the  user  provide  one  estimate 
of  the  representative  workload  for  any  given  time  rather  than  a 
"band"  of  estimates,  as  in  the  probabilistic  approach.  Thus  the 
uncertainty  is  bidden  rather  than  being  made  explicit.  Under  that 
procedure,  a  user  is  forced  to  insert  some  factor  of  safety  in 
making  his  estimate  which  may  be  unduly  high,  since  there  are  pres¬ 
sures  on  him  to  provide  service  to  his  users. 

Providing  only  one  estimate  of  workload  to  the  vendor 
in  the  RFP  does  nor  permit  him  to  perform  suitable  cost-performance 
trade-off  analysis,  since  the  vendor  is  not  given  any  information 
indicating  the  worth  of  excess  system  capacity  to  the  user.  Pro¬ 
viding  the  vendor  with  a  range  of  values  permits  him  to  see  the 
upper  limit  that  has  been  set,  as  well  as  the  estimated  likelihood 
of  reaching  different  workload  levels.  It  thus  permits  the  vendor 
to  design  a  system  capable  of  expanding  to  meet  possible  future 
growth  requirements  and  to  determine  the  worth  of  3uch  an  evolu¬ 
tionary  system  design  in  terms  of  the  costs  and  expectation  of 
using  these  growth  increments.  In  this  way  the  vendor  can  more 
effectively  evaluate  his  alternative  system  configurations  prior 
to  submitting  his  proposal.  This  may  reduce  the  number  of  alter¬ 
native  proposals  which  a  vendor  submits. 

Using  the  proposed  approach  the  source  selection  team 
can  evaluate  the.  vendor  proposal  in  terms  of  its  total  expected 
cost.  By  including  considerations  of  growth  and  determining  their 
cost  implications  rather  than  asking  the  vendor  if  growth  is  avail¬ 
able  but  not  costing  it,  a  more  accurate  estimate  of  the  total  cost 
of  each  vendor’s  proposed  system  can  be  obtained. 

Mandatory  Requirements 

As  discussed  above  in  Section  III.  with  reference  to 
Figure  4,  a  second  part  of  the  selection  process  is  the  satisfying 
of  the  mandatory  requirements.  Each  system  can  be  readily  evaluated 
against  the  mandatory  requirements  since,  by  definition,  all  systems 
must  meet  these  or  the  vendor  is  considered  non -responsive.  For 
this  reason,  when  the.  source  selection  plan  is  constructed,  the  list 
of  mandatory  requirements  should  be  limited  to  those  characteristics 
which  can  be  firmly  defended  on  a  "go/ no-go"  basis.  Any  feature 
which  the  user  desires,  but  cannot  firmly  defend,  should  be  cate¬ 
gorized  as  a  desirable  feature. 
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Desirable  Features 


As  discussed  previously,  the  evaluation  team  must  also 
consider  a  set  of  desirable  features  as  a  hedge  against  uncertainty 
in  the  user's  statement  of  his  expected  workload,  as  a  hedge  against 
the  evaluator's  uncertainty  in  measuring  the  vendor's  capabilities, 
and  as  a  source  of  additional  vendor  capability  that  was  not  ade¬ 
quately  covered  in  the  system  timing. 

a .  Problem  Statement 


There  are  several  reasons  why  the  problem  of  evaluating 
desirable  features  is  a  much  more  difficult  one  than  handling  the 
first  two  elements  of  user  needs,  i.e.,  representative  workload  and 
mandatory  requirements.  First,  it  is  almost  certain  that  each  of  the 
vendors  will  submit  a  different  "mix"  of  desirable  features,  ranging 
from  none  at  all  to  all  of  the  features  requested.  However,  even 
though  two  vendors  submit  the  same  feature,  each  may  have  a  different 
level  associated  with  it,  e.g.,  one  billion  versus  two  billion  char¬ 
acters  of  IAS.  Thus,  the  first  problem  is  how  to  quantitatively 
measure  the  effectiveness  of  the  combination  of  desirable  features 
which  each  vendor  offers.  The  evaluator  can  do  this  by  attempting 
to  determine  the  benefits  to  the  user  jobs  which  each  feature  con¬ 
tributes  and  then  taking  into  account  the  interrelationship  of  sev¬ 
eral  features  as  they  contribute  jointly  to  the  accomplishment  of 
user  jobs.  In  developing  a  method  for  evaluating  a  particular  fea¬ 
ture,  a  way  must  be  found  to  relate  the  characteristics  of  that 
feature  to  the  jobs  whose  performance  will  be  benefited  by  it.  In 
general  the  direct  effect  of  a  system  feature  is  felt  in  the  time 
(system  and/or  staff)  or  quality^of  performing  the  jobs.  If  it  can 
be  determined  that  the  effects  of  a  feature  have  been  adequately 
covered  in  the  estimates  of  system  timing  previously  calculated, 
then  that  feature  need  not  be  considered  separately  in  the  list  of 
desirables.  If  this  is  not  the  case,  then  specific  steps  must  be 
established  for  evaluating  the  feature. 

Even  if  the  evaluator  could  solve  the  first  problem 
of  evaluating  the  benefits  contributed  by  each  desirable  feature, 
the  evaluator  still  has  the  problem  of  determining  if  the  difference 
in  effectiveness  among  vendors  is  worth  the  difference  in  cost. 

Figure  9  illustrates  in  simplified  form  this  problem  of  source 
selection  with  respect  to  desirable  features.  Consider  two  vendors 
each  of  whom  performs  the  future  workload  at  the  same  expected  cost. 
However,  assume  that  vendor  A  has  proposed  a  minimal  EDP  system 
containing  none  of  the  desirable  features  listed  in  the  RFP,  but 
that  vendor  B  has  provided  one  of  the  desirable  features  as  part  of 
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Figure  9.  The  Problem  Of  Evaluating  Desirable  Features 
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his  proposal  at  a  cost  AC  greater  than  vendor  A's.  Assuming  char 
the  performance  of  both  machines  is  identical  in  all  respects  except 
for  this  desirable  feature,  we  could  state  qualitatively  that  the 
effectiveness  of  vendor  B's  system  is  greater  than  vendor  Afs.  How¬ 
ever,  is  the  increased  effectiveness  worth  the  additional  cost  of 
AC  ? 


As  indicated  previously,  the  fundamental  principle  being 
employed  in  the  evaluation  is  to  pivot  on  some  constant  level  of  effec¬ 
tiveness  and  to  choose  the  vendor  who  provides  this  level  of  effective¬ 
ness  at  lowest  cost.  Unfortunately,  the  inclusion  of  desirable  features 
makes  it  difficult  to  define  a  constant  level  of  effectiveness.  The 
user  has  stated  that,  in  addition  to  accomplishing  a  certain  workload, 
he  desires  that  additional  features  also  be  provided.  The  solution  to 
this  problem  can  be  found  in  the  realization  that  if  the  user  desices 
these  features  for  meaningful  reasons,  then  he  must  expect  to  have 
certain  jobs  which  will  benefit  from  these  features.  However,  since 
the  user  has  not  made  the  provision  of  these  features  a  mandatory  re¬ 
quirement,  he  is  implying  that  there  must  be  alternative  ways  of  accom¬ 
plishing  these  jobs  if  the  desirable  feature  is  not  available.  This 
information  enables  the  analyst  to  choose  the  proper  level  of  effective¬ 
ness  for  selection  purposes.  This  will  be  the  level  of  satisfactorily 
performing  the  entire  approved  set  of  user  jobs  in  accordance  witn 
approved  standards  of  performance.  Since  each  desirable  feature  con¬ 
tributes  to  some  of  these  jobs,  it  now  becomes  a  question  of  comparing 
the  proposed  cost  of  any  desirable  feature  against  the  cost  of  other 
alternatives  which  can  be  used  to  do  the  same  job(s).  This  approach 
to  system  selection  translates  the  task  of  evaluating  desirable  features 
to  one  of  cost  analysis.  It  also  leads  to  the  concept  of  the  "worth1' 
of  a  desirable  feature  in  doing  a  job. 

The  term  "worth"  has  been  subjected  to  diverse  economic 
interpretation.  For  our  purposes  we  will  define  the  "worth"  of  a 
feature  in  doing  a  job  as  the  lowest  incremental  cost  to  do  the  same 
job  if  the  feature  is  not  available.  If  the  vendor’s  cost  is  less 
than  the  user’s  "worth",  then  that  feature  will  be  acquired  from  the 
vendor  and  the  cost  will  be  added  to  the  total  system  cost.  If  the 
vendor 3 s  cost  exceeds  the  user's  "worth"  or  if  the  vendor  does  not 
provide  the  feature,  then  the  user  will  make  use  of  an  alternative 
way  and  add  the  corresponding  cost  to  the  vendor’s  total  system  cost. 

If  the  vendor’s  cost  for  the  feature  is  not  separately  identifiable, 
then  there  is  no  way  to  determine  if  his  cost  exceeds  the  user’s 
"worth"  and  his  proposed  cost  will  not  be  cnanged. 

This  process  for  evaluating  desirable  features  will  be 
illustrated  by  an  example  in  Section  III.  A  key  element  in  this 
process  is  the  determination  of  the  "worth"  of  a  desirable  feature. 
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b  Methods  of  Evaluating  Worth 


We  can  distinguish  between  two  basic  ways  for  deter¬ 
mining  the  "worth11  of  the  desirable  feature.  One  method  makes  use 
of  analysis,  the  other  makes  use  of  comparative  ranking  Each  of 
these  methods  will  now  be  examined. 

(1)  Analytical  Determination  of  Worth.  Since  the 
worth  of  a  feature  in  doing  a  job  has  been  defined  as  the  lowest 
incremental  cost  to  do  the  same  job  if  the  feature  is  not  avail¬ 
able,  then  to  determine  the  "worth",  one  must  first  identify  the 
alternative  ways  of  doing  the  job  if  the  feature  were  not  avail¬ 
able,  As  indicated  diagrammatically  in  Figure  10,  there  may  be 
several  ways  of  doing  the  job  without  using  the  feature.  The  cost 
of  each  alternative  way  should  be  determined,  and  then,  making  use 
of  the  probability  that  the  associated  job  will  be  performed,  the 
total  expected  cost  Dver  the  operational  life  of  the  system  should 
be  determined  for  each  of  these  alternative  ways.  The  worth  of  the 
feature  vlII  be  the  least  of  these  costs. 

In  deciding  on  whether  to  utilize  a  particular 
desirable  feature,  the  evaluator  must  also  consider  the  cost  of 
alternative  ways  of  obtaining  the  feature  and  doing  the  job  using 
the  feature.  For  example,  he  might  buy  the  feature  directly  from 
another  source,  Alternatively,  as  for  example  in  the  case  of  a 
software  feature,  the  user  might  develop  the  feature  using  his 
own  resources  (in-house).  If  possible,  then,  the  evaluator  would 
make  his  decision  based  on  doing  the  job(s)  for  which  the  desired 
feature  is  intended  at  lowest'  total  cost.  This  might  be  called 
his  "efficient"  solution. 

Thus  the  efficient  solution  may  be  chosen  by  de¬ 
termining  the  lowest  cost  method  of  obtaining  the  feature  (and 
doing  the  job),  comparing  this  cost  against  the  worth,  and  choosing 
the  lowest,  cost  alternative.  This  process  is  illustrated  in  Figure 
11  This  figure  indicates  diagrammatically  the  cost-effectiveness 
of  two  proposals  both  of  which  perform  the  same  basic  workload  and 
satisfy  the  mandatory  requirements.  These  two  proposals  are  assumed 
to  be  identical  in  all  respects,  differing  only  in  that  one  provides 
a  desirable  feature  F  at  a  total  system  cost  of  C,  whereas  the  other 
does  not.  It  is  immaterial  to  the  present  discussion  whether  these 
proposals  come  from  the  same  or  different  vendors. 

While  the  cost  axis  of  Figure  11  can  be  quantified 
in  a  unidimensional  scale,  the  effectiveness  axis  may  involve  a  num¬ 
ber  of  dimensions  to  represent  the  elements  of  effectiveness.  How¬ 
ever  since  we  are  pivoting  on  the  constant  level  of  effectiveness 
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Figure  10.  Desirable  Features  Alternatives 
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specified  by  the  analyst/user  as  previously  discussed,  we  can  repre¬ 
sent  this  level  diagraramatically  as  shown  in  Figure  11.  This  means 
that  we  are  insisting  that  those  user  requirements,  which  would  bene¬ 
fit  from  the  availability  of  the  feature,  be  satisfied  by  some  other 
means  if  the  vendor  does  not  provide  feature  F.  Analyzing  the  alter¬ 
native  ways  available  to  the  user  results  in  alternatives  and  A2 
with  their  respective  costs  as  shown.  Thus,  by  our  definition,  the 
worth  of  the  feature  is  the  incremental  cost  of  providing  A^  (that  is 

Ci  -  C*p)  . 


However,  it  should  be  noted  that  once  the  worth 
of  a  feature  is  determined  by  analyzing  the  cost  of  a  number  of  dif¬ 
ferent  ways  of  getting  the  job  done  if  the  feature  is  not  used  at 
all,  this  worth  must  be  compared  with  the  cost  of  all  alternative 
ways  of  obtaining  the  feature  and  doing  the  job  using  this  feature. 
Assume  in  the  example  that  the  same  feature  is  available  from  one 
other  source,  labeled  F* ,  and  that  the  job  could  be  done  with  this 
feature  at  a  total  system  cost  of  C  ,  as  shown  in  Figure  11.  The 
proposed  strategy  is  to  choose  the  lowest  cost  method  of  getting  the 
job  done,  whether  by  acquiring  the  feature  or  using  a  lower  cost  al¬ 
ternative.  Hence  in  this  case,  feature  F  would  be  purchased. 

(2)  Determination  of  Worth  by  Comparative  Ranking. 
Sometimes,  because  of  time  and  manpower  limitations,  it  may  not  be 
possible  to  determine  the  worth  of  all  desirable  features  by  analy¬ 
sis  and  considered  judgment.  In  such  cases,  intuitive  judgment  can 
be  used  as  a  part  of  the  quantitative  evaluation.  Such  an  approach 
would  be  implemented  as  follows: 

(a)  Rank  all  of  the  desirable  features  in  order 
of  importance.  The  ranking  should  be  sup¬ 
ported  by  deliberation  utilizing  whatever 
quantitative  analysis  and  considered  judg¬ 
ment  may  be  available. 

(b)  Allocate  points  to  each  feature,  establishing 
its  relative  worth.  Such  relative  worth  will 
be  based  on  the  rationale  developed. 

(c)  Translate  points  to  dollars  of  worth.  This 
is  accomplished  by  calibrating  one  or  more  of 
the  features  through  determining  its  worth  on 
some  analytical  basis  as  previously  described. 
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(d)  Review  the  results  obtained  for  intuitive 

soundness  which  is  the  only  real  test  of  this 
procedure.  If  the  final  results  of  dollar 
worth  of  each  feature  do  not  agree  with  the 
intuitive  feelings  of  the  evaluators  or  source 
selection  plan  reviewers,  an  iteration  of  the 
previous  three  steps  should  be  performed, 
focusing  on  the  following  two  potential  sources 
of  error:  First,  should  the  relative  worth 
(i.e.,  points  assigned)  be  changed?  Second, 
are  there  ways  of  obtaining  the  features  in 
question  at  a  lower  cost  than  the  worth  assigned 
co  the  feature?  Obviously  the  more  features 
that  are  calibrated  by  analysis,  the  more  accu¬ 
rate  will  be  this  procedure. 

c .  Other  Evaluation  Alternatives 


The  most  credible  way  to  evaluate  a  desirable  feature  is 
tc  design  a  live-test  demonstration  that  will  include  the  effects  of 
that  feature.  In  this  way,  the  results  of  the  test  will  provide  an 
explicit  quantitative  measure  of  tne  benefits  of  that  feature.  If 
these  results  can  be  incorporated  into  the  overall  system  timing,  then 
the  particular  feature  need  not  be  given  any  further  separate  considf 
eration.  If  this  is  not  the  case,  then  the  results  of  the  test  may  oe 
used  in  an  evaluation  by  worth  as  described  above. 

However,  one  of  the  practical  constraints  in  an  actual 
siurce  selection  is  the  cost  and  effort  expended  in  the  live-test 
demonstrations.  This  means  that  the  number  of  tests  must  be  kept  at 
a  minimum  with  each  test  designed  to  serve  as  many  testing  functions 
as  possible. 


Under  special  circumstances,  the  following  two  evaluation 
alternatives  may  be  justified: 

(1)  Establish  Design  Goals.  The  user  may  wish  to  estab 
list  a  certain  level  of  hardware  or  software  performance  that  is  char¬ 
acteristic  of  the  present  generation  of  equipment.  If  it  is  difficult 
to  express  the  system  requirements  or  to  design  the  live-test  demon¬ 
stration  in  such  a  way  as  to  rule  out  the  proposal  of  equipment  con¬ 
sidered  by  the  user  to  be  substandard,  then  it  may  be  desirable  and 
justifiable  to  specify  the  feature  as  a  non-discriminating  standard  or 
Hdesign  goal"  which  all  qualified  vendors  can  be  expected  to  meet.  For 
example,  specifying  the  level  of  performance  of  a  card  reader,  card 
punch,  or  printer  might  be  justified  in  this  way. 
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It  should  be  noted  that  if  this  requirement  is  dis¬ 
criminatory  among  the  competing  vendors,  it  would  be  necessary  to 
support  this  requirement  more  carefully  in  terms  of  system  require¬ 
ments  . 


(2)  Qualitative  Evaluation.  If  the  worth  of  a  desirable 
feature  cannot  be  evaluated  quantitatively  by  any  of  the  above  tech¬ 
niques,  a  qualitative  evaluation  should  be  made  and  documented  for  con¬ 
sideration  by  the  Source  Selection  Authority.  Such  qualitative  factors 
would  only  be  considered  and  used  as  "tie -breakers"  if  several  vendors 
were  sufficiently  close  based  on  the  quantitative  evaluation. 

d.  Examples  of  Evaluating  Desirable  Features 

The  following  examples  are  offered  to  illustrate  how 
desirable  features  might  be  evaluated.  It  should  be  emphasized  that 
these  examples  are  only  representative.  Actually  such  features  must 
be  considered  in  the  context  of  the  user’s  system  requirements. 

(1)  Example  1:  Additional  Core  Storage.  The  user  may 
feel  that  additional  jobs  not  included  in  the  representative  workload 
may  occur  which  will  require  additional  core  storage  over  and  above 
that  provided  by  the  vendor  in  meeting  the  basic  workload.  While  the 
best  way  of  measuring  this  feature  would  be  to  include  a  job  requiring 
large  amounts  of  storage  in  one  of  the  benchmark  tests,  it  may  not 
always  be  possible  to  do  this.  Hence,  the  evaluator  must  explore  ways 
of  performing  the  job  if  the  additional  core  storage  were  not  available. 
One  way  of  doing  this  would  be  to  segment  the  job  into  smaller  parts. 

Now,  what  will  this  do  to  system  costs?  First,  programmer  time  will 
increase  due  to  the  additional  programming  load.  Second,  the  system 
running  time  will  increase  due  to  the  lower  efficiency  of  the  operation. 
Both  of  these  will  lead  to  increased  machine  and  staff  time.  The  size 
of  the  increase  will  depend  on  the  complexity  of  the  job  and  the  fre¬ 
quency  of  its  operation.  If  no  other  alternatives  were  available,  the 
cost  of  segmentation  would  be  the  worth  of  this  desirable  feature. 

The  evaluator  must  also  consider  alternative  ways  of 
obtaining  the  feature.  For  example,  it  may  be  possible  to  contract  the 
jobs  requiring  additional  core  storage  to  a  Service  Bureau  or  some  satel¬ 
lite  operation  equipped  to  handle  it,  if  this  is  satisfactory  to  the  user. 
In  this  case,  the  resulting  costs  would  be  estimated  and  the  least  cost 
of  all  alternative  ways  to  obtain  additional  core  storage  would  be  deter¬ 
mined  . 


(2)  Example  2:  Software  Feature.  If  the  job(s)  needing 
the  feature  has  been  included  in  the  live-test  demonstrations,  evaluation 
of  the  feature  is  implicitly  included  in  the  system  timing  obtained,  and 
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another  evaluation  is  not  needed.  If  the  feature  is  not  included 
in  the  live-test,  non-availability  of  the  feature  will  most  likely 
affect  the  programmer  hours  required  to  develop  and  maintain  the 
system  s  programs.  Programmer  hours  would  be  affected  since  the 
programmer  would  now  have  to  do  additional  programming  to  make  up 
for  not  having  the  software  feature  at  all.  One  way  to  handle  this 
would  be  for  the  analyst  to  determine  the  total  number  of  programming 
hours  which  would  be  required  to  do  the  programming  if  the  software 
feature  were  available  at  some  standard  reference  level.  Based  on 
this  reference  level,  one  can  define  the  term  programmer  performance 
factor11  in  the  following  fashion: 


Programmer  Performance  Factor  * 


Reference  lime 
Proposed  Time 


where  the  Reference  Time  is  the  total  estimated  programmer  time  (in 
manhours')  taken  to  program  a  particular  job  using  this  reference  level 
of  software  capability  for  the  feature  in  question,  and  Proposed  Time 
is  the  total  estimated  programmer  time  taken  using  the  vendor  proposed 
level  of  that  feature.  Using  either  live-test  demonstration  results 
or  the  collective  considered  judgment  of  the  evaluators  in  estimating 
the  capability  of  a  software  feature  to  perform  a  certain  class  or 
classes  of  jobs,  the  evaluation  function  illustrated  in  Figure  12  can 
be  constructed.  Making  use  of  this  evaluation  function  and  the  total 
programmer  hours  estimated  by  the  analyst  to  be  required  if  the  soft¬ 
ware  feature  were  available  at  the  reference  level,  the  programmer 
hours  required  for  the  proposed  level  can  be  determined.  For  example, 
if  the  analyst  estimates  that  for  the  reference  level  of  the  software 
feature,  the  programmer  time  would  be  3000  hours,  and  if  he  determines 
that  the  proposed  feature  has  an  efficiency  of  85%,  then  the  estimated 
programmer  hours  required  by  the  vendor1 s  system  is  given  by: 


Estimated  Programmer  Hours 


Reference  Programmer  Hours 

Programmer  Performance  Factor 


3000  Hours 

.85 


3450  Hours 


By  subtracting  the  cost  of  the  estimated  programmer  hours  from  the 
least  costly  method  of  programming  the  job(s)  if  the  desirable  feature 
were  not  available,  the  worth  of  the  feature  can  be  determined. 

Alternative  ways  of  obtaining  the  software  feature 
must  also  be  considered.  For  example,  it  may  be  possible  for  the  user 
to  develop  the  feature  in-house,  requiring  additional  programmer  and 


40 


1.0 


1.00 


PROGRAMMES  PERFORMANCE 
FACTOR  * 8 


.6 


.  A 


.2 


.85 


.70 


.50 


POOR  FAIR  GOOD 

LEVELS  OF  SOFTWARE  CAPABILITY 


EXC, 


Figure  12.  Software  Feature  Evaluation  Function 


machine  time.  Alternatively,  he  may  turn  to  other  software  sources 
and  purchase  the  feature  as  a  package.  The  least  cost  of  these 
alternatives  would  then  be  used  along  with  tne  vorth  in  the  selec  * 
t ion  process . 

(3)  Example  3:  Documentation.  Again  the  question  can 
be  asked,  "How  much  does  it  cost  the  user  if  the  user  is  forced  to 
use  inadequate  document*  at  ion  as  opposed  to  3  Excel  lent’  documentation*" 
Ihe  added  cost  might  be  rhe  extra  time  required  for  readers  of  the 
documentation  as  they  struggle  to  understand  what:  the  author  had  in 
mind.  Thus  documentation  may  be  evaluated  using  the  same  concept,  of 
"efficiency*,  described  previously,  and  calculating  the  larger 
number  of  staff  hours  required,  based  on  the  lDwe.r  efficiency  factor 
due  to  the  unavailability  of  excellent  documentat ion.  If  possible, 
the  analyst  might  include  in  this  determination  of  worth  some  measure 
of  the  co^is  incurred  due  to  system  malfunction  that  might:  occur 
because  of  the  inadequate  documentation.  As  before,  the  analyst 
would  also  want  to  consider  alternative  ways  of  obtaining  excellent 
documentation.  The.  least  cost  of  these  alternatives  would  then  be 
u-ed  along  with  the  worth  in  the  selection  process. 

Note  that  for  this  example,  the.  analyst  might  prefer 
for  various  reasons  (such  as  economy*  to  use  an  alternative  evaluation 
procedure  He  could  handle  documentation  as  a  design  goal  by  requiring 
ir  the  RFP  that  certain  standards  of  documentation  be  satisfied.  In 
chLs  way  the  feature  becomes  a  mandatory  requirement.  Alternatively, 
the  analyst  might  choose  to  process  vendor  differences  in  documentation 
qualitatively  by  noting  the  differences  and  including  the  relevant  in¬ 
formation  for  consideration  only  as  part  of  a  tie -breaking  procedure. 

e .  Remarks 


The  determination  of  the  worth  of  a  desirable  feature  by 
constructing  an  evaluation  function  which  relates  the  performance  bene 
fir.^  of  the  feature  with  cost  is  the  most  satisfactory  way  of  evaluating 
a  desirable  feature  if  its  effects  cannot  be  directly  included  in  system 
timing.  It  should  be  emphasized  that  the  judgment  of  experienced  per  - 
r?-rel  will  be  required  to  construct  the  evaluation  functions.  In  fact, 
the  accuracy  of  the  analysis  is  only  as  good  as  the  experienced  judgment 
and  iubstantiating  data  available.  Undoubtedly  the  most  reliable  sub¬ 
stantiation  would  come  from  benchmark  tests.  While  errors  in  judgment 
are  nevejr  completely  avoidable,  there  are  two  compensating  features  to 
the  ab  :>ve  approach: 

(1)  This  type  of  analysis  forces  the  user  and  evaluator 
co  think  through  and  develop  the  rationale  for  the  need  for  desirable 
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features  and,  hence,  is  superior  to  a  purely  intuitive  judgment  ap¬ 
proach,  since  it  is  more  defendable. 

(2)  The  rationale  and  evaluation  functions  developed 
are  made  explicit  and,  hence,  subject  to  review  by  the  Source  Selec¬ 
tion  Authority.  Thus  they  can  be  changed  if  additional  information 
is  available. 


SELECTION  APPROACH 

Previous  sections  of  this  report  have  indicated  how  to  evaluate 
vendor  proposals  with  respect  to  workload,  mandatory  requirements, 
and  desirable  features.  This  section  will  expand  upon  the  steps  to 
be  followed  in  evaluating  the  vendor  proposals  for  their  desirable 
features  and  in  making  a  final  selection  using  all  of  the  data  gath¬ 
ered.  To  illustrate  the  approach,  a  simplified  example  will  be  used. 

Determining  the  Worth  of  Desirable  Features 

The  Source  Selection  Plan  approved  by  higher  headquarters 
will  include  a  list  of  all  desirable  features  to  be  quantitatively 
evaluated,  the  dollar  worth  (or  evaluation  function  which  describes 
such  worth)  for  each  feature,  as  well  as  the  lowest  known  cost  of 
obtaining  the  feature  separately.  An  example  of  the  worksheet  to  be 
used  in  the  evaluation  (which  can  be  constructed  as  an  appendix  to 
the  Selection  Plan)  is  shown  in  Figure  13,  This  figure  corresponds 
to  an  example  in  which  there  are  three  desirable  features  to  be  con¬ 
sidered,  i.e,,  F^,  F2,  and  F3,  whose  user  worths  and  least  costs  are 
indicated.  (In  the  example  shown,  each  feature  is  either  provided 
completely  or  not  at  all.  If  various  levels  of  a  feature  could  be 
provided,  the  evaluation  function  showing  worth  as  a  function  of 
level  provided  would  be  used  instead  of  the  single  number.) 

Vendor  Submits  Proposed  Casts 

The  proposal  submitted  by  each  vendor  provides  the  following 
information  to  the  evaluators: 

a.  Total  proposed  cost  for  entire  system. 

b.  Cost  of  each  system  and  expansion  capability  required 
to  meet  the  probabilistic  workload, 

c.  Sufficient  information  to  calculate  the  total  expected 
cost  of  performing  the  probabilistic  workload, 

d.  Cost  of  each  separate  desirable  feature  not  included  as 
part  of  the  basic  system. 
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COST  ELEMENTS 

SYSTEM  COST 

1.  TOTAL  PROPOSED  VENDOR  COST 

CA 

s 

cc 

300K 

310K 

330K 

2.  EXPECTED  COST  TO  DO  REPRESENTATIVE  WORKLOAD 

300K 

305K 

310K 

3.  COST  OF  ADDITIONAL  JOB  BENEFITS: 

10K 

10K 

DESIRABLE 

FEATURE 

USER 

WORTH 

LEAST 

COST 

VENDOR  COST 

CA 

CB 

cc 

Fi 

10K 

15K 

— 

lncl. 

15K. 

F2 

25K 

20K 

— 

— 

5K 

2  OK 

20K 

5K 

F3 

10K 

20K 

— 

5K 

lncl. 

10K 

5K 

— 

4.  TOTAL  EXPECTED  COST  TO  DO  USER  JOBS 

340K 

330K 

325K* 

*  Vendor  C  selected  -  lowest  totsl  cost 


Figure  13.  Evaluator's  Worksheet 
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Calculating  Cost  of  Representative  Workload  * 

* 

Utilizing  the  vendor  supplied  information,  the  evaluator 
calculates  the  total  expected  cost  of  each  vendor's  system  to  perform 
the  total  representative  workload  as  previously  described  in  Section 
III.  These  results  are  then  entered  into  the  evaluator's  worksheet 
as  shown  in  Figure  13. 

Validation  of  Mandatory  Requirements 

i 

Based  on  the  vendor  supplied  information,  the  evaluator  must 
validate  that  the  mandatory  requirements  have  been  satisfied. 

Calculating  Cost  of  Additional  Job  Benefits 

The  evaluator  inserts  into  the  Evaluator's  Worksheet  all  of 
the  desirable  features  which  each  vendor  has  proposed  and  the  incre¬ 
mental  costs  associated  with  each  of  these  options.  Note  that  vendor  A 
does  not  provide  any  of  the  three  features,  whereas  the  cost  of  and 
F3  are  included  in  the  cost  of  vendor  B  and  vendor  C,  respectively. 


Based  on  the  cost  information  of  Figure  13,  the  evaluator  can  determine 
for  each  vendor  the  least  costly  of  the  three  alternative  ways  of  re¬ 
ceiving  the  benefits  provided  by  each  of  the  desirable  features.  These 
three  alternatives  are: 

a. 

Buying  the  desirable  feature  from  the  vendor  (at  the 
vendor's  proposed  cost)* 

b. 

Obtaining  the  desirable  feature  from  another  source 
(at  the  least  cost  of  feature  if  obtained  separately) . 

c , 

Not  buying  the  feature,  but  using  the  least  costly 
alternative  way  to  provide  the  benefits  (at  a  cost 
equal  to  user  worth) . 

The  lowest  additional  user  cost  for  obtaining  the  desirable 
feature  (or  its  equivalent)  is  shown  in  Figure  13  as  System  Cost. 

Note  in  the  example  that  the  user  has  stated  that  the  worth  of  F^ 

$10K,  i.e.,  he  can  perform  the  jobs  associated  with  F^  at  an  expected 
cost  of  $10K.  Since  vendor  A  does  not  provide  this  feature,  the  user 
will  be  forced  to  spend  $10K  in  addition  to  vendor  A  costs  to  meet 
those  jobs  associated  with  F^.  Vendor  B  includes  this  feature  as  part 
of  his  asic  system  and  has  stated  that  it  cannot  be  removed  or  priced 
separately.  Hence,  the  user  will  not  have  to  spend  the  $10K  when  using 
vendor  B's  system.  Vendor  C  can  provide  F^  at  a  cost  of  $15K.  Hence, 
the  evaluator  decides  to  eliminate  this  optional  feature  from  vendor  C's 
proposal  since  its  cost  is  higher  than  its  worth  to  the  user,  i.e.,  the 
cost  of  an  alternative  method  for  the  user  to  perform  the  related  jobs. 
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This  same  approach  is  followed  in  determining  which  of  the  other  desir¬ 
able  features  are  to  be  included  in  the  evaluation. 

Calculating  Total  Expected  Cost 

The  total  expected  cost  to  the  user  is  then  calculated  by  add 
ing  the  cost  of  each  desirable  feature  (or  user  cost  equivalent)  to  the 
expected  cost  of  performing  the  probabilistic  Representative  Workload, 
This  total  cost,  shown  in  Figure  13,  completes  tne  cost  calculation. 


Several  interesting  observations  can  be  made  from  analyzing  this 
il  lust  r  at  ion: 

1.  Vendor  A  had  the  lowest  proposed  cost  (since  he  provided  no 
desirable  features)  as  well  as  the  lowest  cost  of  performing  the  repre¬ 
sentative  workload.  On  the  other  hand,  winning  vendor  C  had  the  highest 
proposed  cost  (since  he  had  proposed  all  three  desirable  features)  and 
the  highest  cost  of  performing  the  representative  workload.  However, 
neither  of  these  costs  is  the  proper  measure  for  selection.  If  one 
believes  that  the  user  really  does  have  need  for  the  additional  capa¬ 
bility  represented  by  the  list  of  desirable  features,  and  that  he  will 
have  co  spend  additional  funds  (i.e.,  the  worth)  if  a  desirable  feature 
Is  not  provided,  the  true  criterion  of  choice  must  be  based  on  the  total 
system  costs.  There  were  two  reasons  why  vendor  C  had  the  lowest  total 
cost  in  spite  of  his  other  higher  costs.  First,  he  included  F3  at  no 
additional  cost,  and  this  was  worth  $10K.  Second,  he  provided  F3  for 
$5K  and  the  evaluators  estimated  its  worth  to  be  $20K. 

2,  With  this  approach  there  are  definite  advantages  to  the  vendor 
to  separate  as  many  desirable  features  as  possible  from  the  basic  system 
and  provide  these  as  optional  cost  features  at  a  stated  price  for  each 
feature.  The  reason  for  this  is  that  if  the  calculated  worth  of  each 
feature  is  not  stated  to  the  vendors  (and  it  should  not  be  since  this 
information  may  affect  the  vendor’s  price),  the  vendor  has  no  logical 
way  of  determining  whether  to  propose  a  desirable  feature  or  not.  Hence, 
he  is  forced  to  hedge  his  bets  by  submitting  alternative  proposals,  which 
may  increase  the  vendor’s  proposal  costs  and  the  evaluator’s  selection 
cost-.  However,  with  the  proposed  procedure,  the  vendor  knows  that  the 
evaluator  will  only  choose  those  desirable  features  which  have  value  to 
the  user  and  reject  those  whose  costs  are  too  high.  Hence,  the  vendor 
will  feel  free  to  offer  a  ’’shopping  list”  of  optional  desirable  features, 
each  at  a  separate  price,  as  part  of  his  proposal,  knowing  that  he  can¬ 
not  be  penalized  by  this  strategy. 

in  the  previous  illustration,  if  vendor  C  had  included  the  high  cost 
of  Fi  as  part  of  his  basic  system,  his  total  cost  would  have  been  higher 
and  he  would  have  tied  with  vendor  B, 
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SECTION  IV 


CONCLUSIONS 


SYSTEM  FEASIBILITY 

The  co^t-effectiveness  approach  described  in  this  paper 
appears  feasible  and  a  test  project  to  apply  the  method  to  an  actual 
selection  is  being  carried  out. 

USER  IMPLICATIONS 

This  evaluation  procedure  will  permit  the  user  to  acquire 
a  cost-effective  EDP  system.  It  should  be  emphasized,  however,  that 
additional  analysis  and  data  will  be  required  from  the  user,  relating 
system  specifications  to  its  expected  use,  if  this  procedure  is  to  be 
implemented.  Such  data  will  consist  of  a  representative  workload 
expressed  in  probabilistic  form,  as  agreed  upon  by  the  user  and  AFADA, 
a  set  of  defendable  mandatory  requirements,  and  a  set  of  desirable 
features  together  with  the  worth  to  the  user  for  having  each  feature, 
as  determined  by  the  user  and  ESQ. 

VENDOR  IMPLICATIONS 

This  more  useful  statement  of  user  needs  and  the  selection 
criteria  to  be  used  allows  the  vendors  to  construct  a  better  system 
design  by  performing  more  and  better  cost -performance  trade-off 
analyses,  based  on  better  information.  In  addition,  by  permitting 
the  vendor  to  propose  optional  features,  some  of  which  will  be  se¬ 
lected  by  the  evaluators  on  the  basis  of  its  cost  being  less  than 
its  worth,  the  vendor  may/  not  have  to  propose  separate,  alternative 
proposals,  each  containing  different  combinations  of  desirable  fea¬ 
tures  . 

EVALUATION  IMPLICATIONS 

From  the  evaluator's  point  of  view,  the  proposed  approach  is 
more  defendable  than  other  approaches  examined  for  several  reasons. 
First,  it  is  operationally  oriented  and,  hence,  is  more  rational  and 
should  be  more  understandable  to  reviewers.  Second,  it  avoids  com¬ 
bining  cost  and  performance  factors  which  is  always  difficult  to  jus¬ 
tify,  in  favor  of  choosing  that  system  which  will  satisfy  approved 
user  needs  at  lowest  total  cost  to  the  Air  Force. 
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Since  it  is  more  explicit  and  rational  than  other  pro¬ 
cedures  examined,  it  offers  a  means  of  resolving  differences  of 
opinion  regarding  the  worth  of  system  features.  It  should  be 
stressed  that  the  overall  evaluation  framework  that  has  been 
developed  does  not  eliminate  the  need  for  vendor  validation, 

APPROVAL  AUTHORITY  IMPLICATIONS 

The  proposed  approach  is  consistent  with  OSD  practices 
in  the  field  of  source  selection.  In  addition,  by  quantifying 
the  uncertainty  in  workload,  the  major  factor  in  the  evaluation, 
rather  than  using  a  single  deterministic  estimate,  increased  con¬ 
fidence  in  the  final  selection  is  obtained. 

ADDITIONAL  REMARKS 

This  report  is  intended  to  describe  how  cost-effectiveness 
analysis  can  be  applied  to  EDP  system  selection.  As  the  reader 
will  appreciate,  there  are  many  details  involved  in  the  system 
selection  that  cannot  be  discussed  in  this  report.  Such  details 
are  difficult  to  discuss  in  generality  since  their  relevance  de¬ 
pends  strongly  on  the  context  in  which  they  appear.  We  feel  that 
the  approach  we  have  described  in  this  report  towards  EDP  system 
selection  will  provide  a  formal  framework  into  which  the  details 
of  a  specific  selection  can  be  inserted.  It  is  our  intenrion,  as 
we  gain  experience  in  applying  the  technique,  to  supplement  this 
report  with  a  more  detailed  account  of  how  specific  elements  of  a 
selection  may  be  processed.  At  the  time  of  the  preparation  of  this 
report,  we  have  only  considered  this  technique  in  relation  to  a 
review  of  a  number  of  past  EDP  system  selections.  The  validity  of 
the  technique  can  best  be  established  through  demonstration.  It  is 
our  plan  to  accomplish  this  by  applying  the  technique  to  a  specific 
EDP  system  selection. 

APPLICABILITY  TO  OTHER  SOURCE  SELECTIONS 

One  last  point  of  general  interest  should  be  made  to 
those  readers  who  have  an  interest  in  source  selection  of  systems 
other  than  EDP  systems.  We  have  attempted  to  show  that  the  general 
principles  of  cost-effectiveness  analysis  which  have  been  applied 
so  often  to  the  concept  formulation  or  systems  planning  phase  of 
the  systems  acquisition  process  can  also  be  applied  to  the  source 
selection  process,  specifically  of  EDP  systems.  The  same  approach 
can  also  be  applied  to  other  systems.  In  fact,  it  may  be  easier 
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to  apply  this  to  other  areas  where  the  measure  of  effectiveness 
is  more  easily  defined  and  measured.  For  this  reason,  evaluators 
in  other  system  areas  should  give  thought  to  the  possibility  of 
applying  the  cost-effectiveness  approach  in  a  form  tailored  to 
their  problem,  as  was  done  for  the  EDP  problem. 
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